WASHINGTON — The U.S. Environmental Protection Agency’s current framework for evaluating the scientific evidence used for setting National Ambient Air Quality Standards (NAAQS) is effective, reliable, and scientifically defensible provided that key scientific questions are identified and a range of appropriate expertise is engaged, says a new report from the National Academies of Sciences, Engineering, and Medicine. The framework uses a “weight-of-evidence” approach, which allows EPA to integrate scientific findings by combining assessments of the quality of studies with expert judgment.

More transparency in how evidence is integrated could improve confidence in EPA’s determinations, including its guidance on how to examine evidence for key sensitive groups — such as older or immunocompromised people, or endangered species or habitats, the report says.

Under the Clean Air Act, EPA is required to set NAAQS for the following air pollutants: carbon monoxide, lead, oxides of nitrogen, particulate matter, ozone and related photochemical oxidants, and sulfur dioxide. As one step in a multistep review process, EPA’s Integrated Science Assessment (ISA) informs the NAAQS by applying a causal determination framework that guides evaluation of the strength of the evidence on whether the pollutants cause adverse impacts for human health and public welfare — which includes impacts on wildlife, water, forests, agriculture, and climate.

EPA asked the National Academies to examine available methods and frameworks for assessing causality, to consider new advances and tools for integrating and evaluating evidence, and to recommend how such methods and tools might be incorporated into a future causality determination framework. The committee that conducted the study was not asked to review the ISA process, the NAAQS, or the process for setting the NAAQS.

“Determining causal relationships in complex air pollution settings is challenging, and in general, many studies are needed — each with strengths and weaknesses,” said Armistead G. Russell, the Howard T. Tellepsen Chair and Regents’ Professor of Civil and Environmental Engineering at Georgia Institute of Technology, and co-chair of the committee that wrote the report. “After carefully examining the effectiveness of numerous frameworks and tools, our committee found that EPA’s current weight-of-evidence approach to evaluate these studies and make determinations is scientifically defensible and reasonable.”

EPA’s causality framework categorizes the evidence on the air pollutants as follows: “causal relationship,” “likely to be a causal relationship,” “suggestive of, but not sufficient to infer, a causal relationship,” “inadequate to infer the presence or absence of a causal relationship,” and “not likely a causal relationship.” The current number and description of EPA’s causal categories are scientifically meaningful and useful, the report says, and help EPA determine which health or public welfare effects should be carried into the risk assessment and exposure stage of setting the NAAQS.

A single framework is appropriate to assess the causality of both human health and public welfare effects of the pollutants as long as causal determinations are adequately supported and explained and the significance of relevant exposure patterns and welfare endpoints is made clear, the report says. “Core scientific principles cut across studies, whether they are studying human health or public welfare outcomes,” said committee co-chair Elizabeth A. Stuart, executive vice dean for academic affairs and Bloomberg Professor of American Health at Johns Hopkins Bloomberg School of Public Health. “Although the studies may differ substantially, questions of exposure assessment, study design and quality, and transparency and replicability are equally relevant.” Using a single framework allows for a uniform approach across study designs and disciplines and enhances the ability to examine and integrate the linkages between human health and public welfare effects.

The report made a number of recommendations for EPA to improve its causality framework:

• When practical, the framework should provide guidance on how evidence should be examined for key sensitive groups to ensure that causal determinations fully account for effects in these groups. Currently, the framework provides guidance on how these groups are handled in the assessments only after a causal determination is made, which may mask potential causal impacts for sensitive populations.
• Provide explicit guidance for evaluating approaches used in studies to account for confounding effects — such as other pollutants, weather effects, or socioeconomic or demographic factors — and how these approaches could influence weight-of-evidence considerations.
• Develop guidance for assessing data, methods, and assumptions in individual studies, and for how each study informs the weight-of-evidence approach. Attention to transparency and reproducibility is particularly important for complex study designs.
• Clearly articulate a process for identifying and including the necessary expertise and perspectives for each step of causal determination, from development to study selection and assessment to final review.

• When practical, the framework should provide guidance on how evidence should be examined for key sensitive groups to ensure that causal determinations fully account for effects in these groups. Currently, the framework provides guidance on how these groups are handled in the assessments only after a causal determination is made, which may mask potential causal impacts for sensitive populations.
• Provide explicit guidance for evaluating approaches used in studies to account for confounding effects — such as other pollutants, weather effects, or socioeconomic or demographic factors — and how these approaches could influence weight-of-evidence considerations.
• Develop guidance for assessing data, methods, and assumptions in individual studies, and for how each study informs the weight-of-evidence approach. Attention to transparency and reproducibility is particularly important for complex study designs.
• Clearly articulate a process for identifying and including the necessary expertise and perspectives for each step of causal determination, from development to study selection and assessment to final review.

The study — undertaken by the Committee on Assessing Causality from a Multidisciplinary Evidence Base for National Ambient Air Quality Standards — was sponsored by the U.S. Environmental Protection Agency.

The National Academies of Sciences, Engineering, and Medicine are private, nonprofit institutions that provide independent, objective analysis and advice to the nation to solve complex problems and inform public policy decisions related to science, technology, and medicine. They operate under an 1863 congressional charter to the National Academy of Sciences, signed by President Lincoln.

Contact:
Megan Lowry, Media Officer
Office of News and Public Information
202-334-2138; e-mail news@nas.edu